Robotic tray gripper

ABSTRACT

A robotic end effector is disclosed. The end effector is configured to grasp, move, and place one or more objects without assistance from another robot. The end effector includes a lateral member configured to be coupled to a robotic arm, a passive side member coupled substantially rigidly to the lateral member at a first distal end and configured to engage mechanically with a first recess on a first side of an object to be grasped, and an active side member coupled to the lateral member at a second distal end opposite the first distal end and configured to engage mechanically with a second recess on a second side of the object to be grasped, the second side being opposite the first side of the object to be grasped. The passive side member and the active side member each include a structure, and the structure on the passive side member and the active side member have different profiles.

CROSS REFERENCE TO OTHER APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 63/175,784 entitled ROBOTIC TRAY GRIPPER filed Apr. 16, 2021 whichis incorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

In certain warehouse and similar operations, a set of tasks sometimesreferred to herein as “line kitting” may be performed to assemblestacked trays of items for further distribution, such as delivery to aretail point of sale. Stacks of trays containing the same type of itemmay be received, and trays may be drawn from different homogeneousstacks each having trays of items of a corresponding type to assemble amixed stack of trays, e.g., to be sent to a given destination.

For example, a bakery may bake different types of products and may fillstackable trays each with a corresponding homogeneous type of product,such as a particular type of bread or other baked good. Stacks of traysmay be provided by the bakery, e.g., to a distribution center. One stackmay include trays holding loaves of sliced white bread, another may havetrays holding loaves of whole wheat bread, still another tray holdingpackages of blueberry cupcakes, etc. Trays may be drawn from the variousstacks to assemble a (potentially) mixed stack of trays. For example, astack of six trays of white bread, three trays of whole wheat, and onetray of blueberry cupcakes may be assembled, e.g., for delivery to aretail store.

While the above example involves trays of different types of baked good,in other line kitting operations stackable trays may hold otherproducts.

In a typical approach, trays are handled by human workers. The trays mayinclude handholds to enable a human worker to grasp and move trays,e.g., by placing the workers hand on or in the handhold. Such work byhuman workers may cause fatigue or injuries, may take a lot of time tocomplete, and could be error prone.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1A is a block diagram illustrating an embodiment of a robotic linekitting system.

FIG. 1B is a block diagram illustrating an embodiment of a robotic linekitting system.

FIG. 2A is a state diagram illustrating an embodiment of an automatedprocess to assemble stacks of trays.

FIG. 2B is a flow diagram illustrating an embodiment of an automatedprocess to assemble stacks of trays.

FIG. 3A is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 3B is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 3C is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 3D is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 3E is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 4A is a flow diagram illustrating an embodiment of a process forgrasping one or more tray using a robotically controlled tray handlingend effector.

FIG. 4B is a flow diagram illustrating an embodiment of a process forgrasping one or more tray using a robotically controlled tray handlingend effector.

FIG. 4C is a flow diagram illustrating an embodiment of a process forgrasping one or more tray using a robotically controlled tray handlingend effector.

FIG. 4D is a flow diagram illustrating an embodiment of a process forreleasing one or more tray using a robotically controlled tray handlingend effector.

FIG. 5A is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 5B is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 5C is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 6A is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 6B is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 6C is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 6D is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 6E is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 6F is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 6G is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 7A is a diagram illustrating an embodiment of a sensor for arobotically controlled tray handling end effector.

FIG. 7B is a diagram illustrating an embodiment of a sensor for arobotically controlled tray handling end effector.

FIG. 8A is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 8B is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector.

FIG. 9A is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector equipped with a guide fin.

FIG. 9B is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector equipped with a guide fin.

FIG. 9C is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector equipped with a guide fin.

FIG. 9D is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector equipped with a guide fin.

FIG. 10 is a flow diagram illustrating an embodiment of an automatedprocess to place one or more trays on a stack.

FIG. 11 is a diagram illustrating an example of a stack of traysconfigured to be stacked in a specific tray orientation.

FIG. 12 is a diagram illustrating an embodiment of a tray handlingrobot.

FIG. 13 is a diagram illustrating an embodiment of a tray handlingrobot.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

An autonomous, tray handling, line kitting robot is disclosed. Invarious embodiments, a line kitting robot as disclosed herein includes arobotic arm having an end effector as disclosed herein comprisingstructures to grasp a tray of items. In various embodiments, one or moresuch robots may operate together in a single workspace to grasp traysfrom source stacks and move them, singly or in groups, to destinationsstacks assembled according to invoice, manifest, or other input dataindicating the output stacks required to be assembled. In someembodiments, two or more robots as disclosed herein may operate on thesame rail or other transport structure. Operation is coordinated toavoid collisions and to make efficient use of all robots to complete anoverall set of line kitting tasks, such as to assemble a plurality ofoutput stacks each having a corresponding mix of trays according toinput information, such as invoices, manifests, etc.

Various embodiments include an end effector device (also referred toherein as an end effector) to be comprised in, or connected to, arobotic arm to grasp, move, and place one or more trays. The endeffector comprises (i) a lateral member configured to be attached to afree moving end of a robotic arm, (ii) a passive side member coupled tothe lateral member at a first distal end and configured to engagemechanically with a first structure on a first side of a tray, (iii) anactive side member coupled to the lateral member at a second distal endopposite the first distal end and configured to engage mechanically witha second structure on a second side of the tray opposite the first sideof the tray, and (iv) an active side actuation device comprising astationary element coupled to the lateral member and an active elementcoupled to the active side member at or near an upper end of the activeside member in a manner such that actuation of the active side actuationdevice moves the active element causing the active side member to rotateabout a rotation axis into a position in which the active side member issubstantially parallel to the lateral side member. In some embodiments,the passive side member comprises a passive-side structure that isconfigured to engage a hole, a recess, or a handle comprised in thefirst structure, and the active side member comprises an active-sidestructure that is configured to engage a hole, a recess, or a handlecomprised in the second structure. In some embodiments, a first profileof the passive-side structure is different from a second profile of theactive-side structure.

Various embodiments comprise a tray handling robotic system. The trayhandling robotic system comprises (i) a memory configured to store dataindicating a set of output stacks to be assembled, each output stackincluding an associated set of trays, and (ii) a processor coupled tothe memory and configured to control operation of one or more robots,each configured to grasp, move, and place one or more trays at a time,according to a plan, to iteratively pick trays from source stacks oftrays and assemble the set of output stacks, including by building eachoutput stack by successively placing on the output stack trays pickedfrom one or more corresponding source stacks. Each of the robotscomprises a robotic arm and a tray handling end effector configured tograsp, move, and place one or more trays without assistance from anotherrobot. The end effector comprises (i) a lateral member configured to beattached to a free moving end of a robotic arm, (ii) a passive sidemember coupled to the lateral member at a first distal end andconfigured to engage mechanically with a first structure on a first sideof a tray, (iii) an active side member coupled to the lateral member ata second distal end opposite the first distal end and configured toengage mechanically with a second structure on a second side of the trayopposite the first side of the tray, and (iv) an active side actuationdevice comprising a stationary element coupled to the lateral member andan active element coupled to the active side member at or near an upperend of the active side member in a manner such that actuation of theactive side actuation device moves the active element causing the activeside member to rotate about a rotation axis into a position in which theactive side member is substantially parallel to the lateral side member.In some embodiments, the passive side member comprises a passive-sidestructure that is configured to engage a hole, a recess, or a handlecomprised in the first structure, and the active side member comprisesan active-side structure that is configured to engage a hole, a recess,or a handle comprised in the second structure. In some embodiments, afirst profile of the passive-side structure is different from a secondprofile of the active-side structure.

Various embodiments include a robotic end effector. The robotic endeffector includes a lateral member configured to be coupled to a roboticarm, a passive side member coupled substantially rigidly to the lateralmember at a first distal end and configured to engage mechanically witha first recess on a first side of an object to be grasped, and an activeside member coupled to the lateral member at a second distal endopposite the first distal end and configured to engage mechanically witha second recess on a second side of the object to be grasped, the secondside being opposite the first side of the object to be grasped. Thepassive side member comprises a passive-side structure that isconfigured to be inserted into the first recess. The active side membercomprises an active-side structure that is configured to be insertedinto the second recess. A first profile of the passive-side structure isdifferent from a second profile of the active-side structure.

In various embodiments, a tray handling robotic system as disclosedherein includes a single rail system occupied by multiple robotscoordinating the fulfilment of trays containing packaged food goods, orany other commercial goods or other items. The trays may arrive instacks of various heights and stacked in various orientations. In someembodiments, the system is divided into two sides: an input side wherehomogenous stacks come in and an output side that is dedicated tovarious customers and/or other destinations and is formed by kittingvarious products from the input side based on an order list, forexample.

In some embodiments, multiple robots operate on the same rail or othertransport system. For example, two or more robots may operate on thesame rail system. Each robot is mounted on a chassis that can be movedalong the rail under robotic control, independently of each other robot.The robots are aware of each other and coordinate their motions tooptimize order fulfillment. Each robot may use of a single tray gripperdesigned to grasp trays. Alternatively, a robot may use a tray gripperthat is designed to grasp a plurality of trays at one time. In variousembodiments, the gripper is modular and can be adapted to a variety ofdifferent trays.

Various embodiments improve control of an end effector in connectionwith grasping, moving, and placing one or more trays. The end effectorincludes a set of sensors used to improve detection of a position of atleast part of the end effector and/or detection with respect to graspinga tray. For example, one or more sensors comprised in a passive sidemember (also referred to herein as a passive arm) and/or one or moresensors comprised in an active side member (also referred to herein asan active arm) are used in connection with detecting whether a tray isgrasped by the end effector, or otherwise determine a position of theend effector (or at least part of the passive arm or at least part ofthe active arm) relative to a tray.

FIG. 1A is a block diagram illustrating an embodiment of a robotic linekitting system. In the example shown, system 100 includes source traystacks 102 and 104 moving along an input stack conveyance (e.g.,conveyance 106) fed in this example from an input end 108 (staging andloading area). Each of the source tray stacks 102 and 104 in thisexample is shown to be stacked on a wheeled cart or chassis. In variousembodiments, the source tray stacks 102 and 104 may be pushed manuallyonto the conveyance 106, which may be a conveyor belt or other structureconfigured to advance the source tray stacks 102 and 104 through theworkspace defined by conveyance 106. In some embodiments, the chassis orother base structure on which the source trays are stacked may beself-propelled. In some embodiments, source tray stacks 102 and 104 maybe advanced through/by conveyance 106 under robotic control. Forexample, the speed and times at which the source tray stacks 102 and 104are advanced by/through conveyance 106 may be controlled to facilitateefficient grasping of trays from the source tray stacks 102 and 104.

In the example shown, a single rail (e.g., rail 110) is disposed alongone long side of the conveyance 106. In this example, two robots, onecomprising robotic arm 112 and another comprising robotic arm 114, aremounted movably, independent of one another, on rail 110. For example,each robotic arm 112, 114 may be mounted on a self-propelled chassisthat rides along rail 110. In this example, each robotic arm 112, 114terminates with a tray handling end effector (e.g., end effector 116,118). In some embodiments, end effector 116 and/or 118 implements endeffector 300 of FIGS. 3A-3D, end effector 300 of FIGS. 5A-5C, endeffector 600 of FIGS. 6A-6E, end effector 800 of FIGS. 8A-8D, and/or endeffector 1204 of FIG. 12.

In various embodiments, the tray handling end effector (e.g., endeffector 116 or 118) is operated under robotic control to grasp one ormore trays from a source tray stack 102, 104. As shown in FIG. 1A, eachend effector 116, 118 includes a lateral member attached to the end ofthe robotic arm 112, 114. A side member is mounted on each end of thelateral member. As shown, at least one of the side members is opened orclosed under robotic control, in various embodiments, to enable a trayto be grasped (by closing the side member) or released (by opening theside member).

In various embodiments, each end effector 116, 118 includes onenon-moving (“passive”) side member and one movable (“active”) sidemember. In this example, the movable or “active” side member swings open(position in which end effector 116 is shown), e.g., to enable the endeffector to be placed in position to grasp one or more trays, and swingsclosed (position in which end effector 118 is shown), e.g., to completea grasp of one or more trays. In various embodiments, a robotic controlsystem (e.g., a computer that controls robotic arm 112, 114, such ascontrol computer 128) controls the end effector to actuate theopening/closing of the end effector such as in connection with graspingor releasing a tray. The robotic control system controls the endeffector based at least in part on image data of the work space and/orone or more sensors comprised in (or connected to) the corresponding endeffector. In some embodiments, the one or more sensors one or moresensors comprised in (or connected to) the corresponding end effectorare configured to: (i) obtain information indicative of whether agasping mechanism (e.g., an active member) of the end effector is in anopen position or a closed position, (ii) obtain information indicativeof an extent to which the grasping mechanism is open, (iii) obtaininformation indicative of when the tray (or end effector relative to thetray) is in a position at which the end effector is controlled to engageat least one side of the end effector (e.g., a passive member or astructure comprised on the passive member) with a hole, a recess, or thea comprised in a side of a tray (e.g., a tray being grasped), (iv)obtain information indicative of when the tray (or end effector relativeto the tray) is in a position at which the end effector (e.g., a passivemember or a structure comprised on the passive member) is engaged withthe hole, the recess, or the handle comprised in the a side of a tray,and/or (v) obtain information indicative of whether the graspingmechanism is closed or otherwise engaged with the tray.

In various embodiments, each end effector 116, 118 includes on each sidemember one or more protrusions or similar structures of a size and shapesuch that the protrusion, etc., fits into and, in various embodiments,can be slid under robotic control into holes or other openings in thesides the tray(s) to be grasped. For example, in some embodiments,protrusions on the inner face of the side members, sometimes called“thumbs” herein, may be slotted into handholds (e.g., holes sized toaccommodate a human hand) on opposite sides of a tray, as described andillustrated more fully below.

In various embodiments, the respective robotic arms 112, 114 areoperated at the same time, fully autonomously, to pick trays from sourcetray stacks 102, 104 and place them on destination tray stacks, such asdestination tray stacks 120, 122, in a destination tray stack assemblyarea on an opposite side of rail 110 from conveyance 106 and source traystacks 102, 104. The destination tray stacks may be assembled, invarious embodiments, according to invoice, manifest, order, or otherinformation. For example, for each of a plurality of physicaldestinations (e.g., retail stores), a destination stack associated withthat destination (e.g., according to an order placed by the destination)is built by selecting trays from respective source tray stacks 102, 104and stacking them on a corresponding destination tray stack 120, 122.Completed destination tray stacks 120, 122 may be removed from thedestination tray stack assembly area, as indicated by arrow 124, e.g.,to be place on trucks, rail cars, containers, etc. for delivery to afurther destination, such as a retail store.

Referring further to FIG. 1A, in the example shown in the system 100includes a control computer 128 configured to communicate wirelesslywith robotic elements comprising system 100, including in variousembodiments one of more of conveyance 106; the wheeled chassis on whichsource tray stacks 102, 104 are stacked (if self-propelled); the roboticarms 112, 114 and/or the respective chassis on which the robotic arms112, 114 are mounted on rail 110; and the robotically controlled trayhandling end effectors (e.g., end effectors 116, 118). In variousembodiments, the robotic elements are controlled by control computer 128based on input data, such invoice, order, and/or manifest information,as well as input state information, such inventory data indicating whichsource tray stacks include which type and/or quantity of product.

In various embodiments, source tray stacks 102, 104 may be inserted intoa gate or other ingress/control structure at the input end 108 ofconveyance 106. Conveyance 106 may comprise an apparatus (stack mover)that moves the source tray stacks 102, 104 along the rail 110 tooptimize throughput and minimize robot displacement, e.g., by minimizinghow far and/or often the robotic arms 112, 114 must be moved along rail110 to grasp source trays and place them on respective destinationstacks. The source tray stacks 102, 104 can come in with trays indifferent orientations/weights/and weight distribution. The system 100uses force and moment control to operate robotic arms 112, 114 to inserta thumb or other protrusion gently and securely into a tray and plansits motion and tray trajectory in order to not collide with itself orthe environment. In various embodiments, each robotic arm 112, 114operates in a very tight space of roughly 2.5 m in width and has a verylight footprint. The robot utilizes its full workspace and intelligentlyplans its motion optimizing its grasp. It recognizes the need to performorientation changes and handles that accordingly while avoidingobstacles. The robot moves to the correct output (e.g., destination traystack 120, 122) corresponding to the right customer while coordinatingwith the other robots on the rail 110. It then uses advanced forcecontrol and interactions with the environment to figure out a properplace strategy. The cycle then restarts.

In the example shown in FIG. 1A, the system 100 includes a 3D camera126. In various embodiments, the system 100 may include a plurality of3D (or other) cameras, such as camera 126, and may use image and depthdata generated by such cameras to generate a three-dimensional view ofat least relevant portions of the workspace and scene, such as thescene/state shown in FIG. 1A. In some embodiments, cameras such ascamera 126 may be used to identify the contents of trays in source trayscomprising a tray stack, e.g., by recognizing the size, shape,packaging, and/or labeling of such items, and/or by recognizing theshape, color, dimensions, or other attributes of the source stack traysthemselves and/or by reading bar code, QR code, radio frequency tag, orother image or non-image based information on or emitted by the trays.

In various embodiments, image data generated by cameras such as camera126 is used to move robotic arms and end effectors into a position neara tray or stack of two or more trays to be grasped and picked up from asource stack and/or to position the tray(s) near a destination at whichthey are to be place, e.g., at the top of a corresponding destinationstack. In some embodiments, force control is used, as described morefully below, to complete the final phases of a pick/grasp episode and/ora placement episode.

Although a single camera (e.g., camera 126) mounted to a wall in theworkspace of system 100 is shown in FIG. 1A, in various embodiments,multiple cameras or other sensors, or a combination thereof, may bemounted statically in a workspace. In addition, or instead, one or morecameras or other sensors may be mounted on or near each robotic arm 112,114, such as on the arm itself and/or on the end effector 116, 118,and/or on a structure that travels with the robotic arm 112, 114 as itis moved along rail 110.

FIG. 1B is a block diagram illustrating an embodiment of a robotic linekitting system. In FIG. 1B, an example is shown of an overhead view of aworkspace in which the system 100 of FIG. 1A may operate. In the exampleshown, robotic arms 112, 114 move along a common rail (e.g., rail 110),as in FIG. 1A, to access and pick trays from source stacks 140 movingalong conveyance 106 and play trays on corresponding destination stacks142 in the destination stack assembly area on the opposite side of rail110 from the source stacks 140 and conveyance 106. In this example, ahuman worker manually feeds source stacks onto the conveyance 106, butin some embodiments a robotic worker performs all or part of that task,e.g., according to plan generated programmatically to fulfill a set oforders, each associated with a corresponding destination. Asdestinations stacks 142 are completed, they are moved out of thedestination stack assembly area, as indicated by the arrows that the topof FIG. 1B, which corresponding to arrow 124 of FIG. 1A.

While in the example shown in FIGS. 1A and 1B the trays each containonly one type of time, in other embodiments and applications source anddestination trays having mixes of items may be handled to assembledestination stacks of trays as disclosed herein. Similarly, while in theexample shown in FIGS. 1A and 1B the source stacks of trays each containonly trays of the same type and content, in other embodiments andapplications source tray stacks may include a mix of trays and/or itemtypes. For example, the control computer 128 may be provided withinformation indicating which types of tray are in which position in eachsource tray stack, and may use that information, along with manifest orother information indicating the required contents of each destinationtray stack, to build the required destination tray stacks by pickingneeded trays each from a corresponding position on a source tray stackand adding the tray to a corresponding destination stack.

FIG. 2A is a state diagram illustrating an embodiment of an automatedprocess to assemble stacks of trays. In various embodiments, processingaccording to the state diagram 200 is performed by a control computer,such as control computer 128 of FIG. 1A. In the example shown, aplanning state, process, and/or module 202 generates and dynamicallyupdates a plan to assemble output stacks of trays by using roboticinstrumentalities as disclosed herein to pick trays from homogeneous ornon-homogeneous source stacks of trays and building destination stackseach having one or more types of trays, e.g., according to a set oforders, invoices, manifests, etc. The planning state, process, and/ormodule 202 receives feedback indicating which destination tray stackshave been completed, which stacks of source trays have been moved intothe workspace, and/or other state and context information which can beused to continuous update the plan to pick and place (stack) trays toassemble the destination stacks. In state 204, a process controlling agiven robotic instrumentality (e.g., robotic arms 112 and/or 114 andassociated end effectors 116 and 118, in the example shown in FIG. 1A)determines a next set of one or more trays to be move from a sourcestack to a destination stack according to a current overall plan asreceived from planning state, process, and/or module 202. For example,the robot may determine to grasp one, two, or more trays from a sourcestack to add them to (or start a new) destination stack. The robotenters state 206, in which a strategy and plan may be determined to doone or more of move into position to grasp the tray(s), grasp the trays,and/or begin to move them toward the destination stack location isformed; and the robot moves into position and grasps the trays. Once thetray(s) has/have been grasped, the robot enters state 208 in which thetray is moved along a planned (and, if needed, dynamically adapted)trajectory to the vicinity of the destination stack, e.g., a positionhovering over the destination stack and/or a location or structure onwhich the destination stack is to be built. In state 210, the robotplace(s) the tray(s) on the destination stack. In some embodiments, thestate 210 includes maneuvers under force control to verify the tray(s)is/are place securely on the destination stack, e.g., by moving (orattempting to move) the tray(s) forward and backward (or side to side,as applicable) to ensure any interconnecting structures are aligned andwell slotted, such as tabs on the bottom of the trays being placedfitting into corresponding recesses in the side walls of the tray onwhich the tray(s) is/are being placed. Once the trays are determined tohave been placed securely, the robot releases the tray(s) and reentersthe state 204, in which a next set of one or more trays is determined tobe picked from a corresponding source stack and moved to a correspondingdestination stack, e.g., according to overall plan information receivedfrom planning state, process, and/or module 202. In various embodiments,a robotic system as disclosed herein continues to cycle through thestates 204, 206, 208, and 210 of FIG. 2A until all destination stackshave been assembled.

FIG. 2B is a flow diagram illustrating an embodiment of an automatedprocess to assemble stacks of trays. In various embodiments, a processor module controlling one or more tray handling robots implements theprocess 220 of FIG. 2B. In various embodiments, the process 220 of FIG.2B is performed by process or module running on a control computer, suchas control computer 128 of FIG. 1A. In the example shown, at 222 aspecific set of one or more trays is determined to be moved from asource stack to a destination stack. In some embodiments, a robotic armhas an end effector that accommodates picking and placing only one trayat a time. In other embodiments, a robot has an end effector that cangrasp a stack of two or more trays, e.g., by grasping a bottommost oneof the trays in the stack to be grasped. At 224, a strategy to move toand grasp the tray(s) is determined. For example, the robot may plan andimplement a set of maneuvers to move its end effector to a positionabove or otherwise near the tray(s) to be grasped. A strategy to graspthe tray(s) is determined and implemented. At 226, a plan (e.g.,trajectory) to move the tray(s) to a destination stack is determined andexecuted. The trajectory/plan takes into consideration obstacles in theworkspace, such as other stacks, and potential conflicts with otherrobotic instrumentalities, such as another pick/place robot operating inthe same workspace (e.g., robotic arms 112, 114 of FIG. 1A). At 228, astrategy to place the tray(s) atop the corresponding destination stackis determined and executed. At 230, results of the pick/place operationare reported, e.g., to a planning process or module. Subsequentiterations of steps 222, 224, 226, 228, and 230 are repeated until it isdetermined at 232 that processing is done, e.g., all destination stackshave been completed.

FIG. 3A is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. In the example shown, endeffector 300 includes a lateral member 302 to which a passive sidemember 304 is fixedly mounted and an active side member 306 is hinge orotherwise movably mounted in a manner that enables active side member306 to be moved to an open position that facilitates moving the endeffector 300 into a position to grasp a tray. An active side thumb 308is positioned on (or comprises an integral part or feature of) an innerface of active side member 306.

According to various embodiments, active side member 306 is movablewithin a predefined range of motion. As an example, end effector 300includes one or more stopping mechanisms (e.g., stopper, switch, or thelike, or a combination thereof) that restrict movement of the activeside member 306 to within the predefined range of motion. End effector300 may include an open-position stopping mechanism that prevents activeside member 306 from moving in an opening direction past an openposition threshold (e.g., 130 degrees relative to a plane/vector alongwhich lateral member 302 extends in a lengthwise direction, or between30 and 50 degrees relative to a closed position at which active member306 is substantially normal to the plane/vector along which lateralmember 302 extends). End effector 300 may include a closed-positionstopping mechanism that prevents active side member 306 from moving inan closing direction past a closed position threshold (e.g., about 90degrees relative to a plane/vector along which lateral member 302extends in a lengthwise direction, etc.). Various values can be selectedfor the open position threshold and/or the closed position threshold. Insome embodiments, the open position threshold is set based at least inpart on an environment in which the robot to which end effector 300 isconnected operates. As an example, if a plurality of robots areoperating within a relatively close proximity, the range of motion ofthe active side member 306 is based at least in part on a distancebetween robots or between zones in which the various robots (e.g.,neighboring robots) operate. As the active side member 306 moves from aclosed position to an open position the further the active side member306 extends in the x-direction. In addition, the further the active sidemember 306 is movable from the closed position to the open position, thegreater the time required for the robotic system to control toopen/close active side member 306 in connection with grasping/placing atray(s). Accordingly, limiting the range of motion of the active sidemember 306 (e.g., to a sufficient open position threshold to permit theend effector to grasp a set of one or more tray(s) with ease) allows therobotic system to operate more efficiently within proximity of otherrobots (e.g., other robots that are autonomously grasping, moving, andplacing trays).

In some embodiments, the open position threshold and/or the closedposition threshold are configurable. For example, the one or morestopping mechanisms are configurable and set based on the desired theopen position threshold and/or the closed position thresholdconfiguration(s).

The active side thumb 308 and a corresponding structure on the innerface of passive side member 304, not visible in FIG. 3A, are of a sizeand shape suitable to be inserted into a handhold or other recess orhole on opposite sides of a tray to be grasped by the end effector 300.In various embodiments, the thumbs 308 are removable and replaceable,e.g., to be replaced once they are worn out from use or to be exchangedwith a thumb having a different shape, dimensions, materials, etc.suitable to grasp a different type of tray, for example. Active sidethumb 308 is fixedly mounted to active side member 306 such as to impedethumb 308 from rotating (e.g., during engagement with tray handle,etc.). For example, active side thumb is mounted to active side member306 at three mounting points. Various other mounting configurations ornumber of mounting points may be implemented. As shown in the three-viewdrawing to the right of FIG. 3A, in the example shown the thumb 308 hasconvex surfaces 308 a-d on each of four sides. In various embodiments,the convex surfaces 308 a-d facilitate using force and moment control toinsert the thumb 308 into a handle or other hole or recess in the sideof a tray to be grasped. In some embodiments, the convex surfaces areused in conjunction with active force control and orientation impedancecontrol to ensure a gentle and secure final grasp, where the active sideis fully into the tray. For example, even if imperfectly aligned, aconvex surface 308 a-d engaged in a side or edge of a hole may enablethe rest of the thumb 308 to more readily be slid more fully into thehole. Flat surfaces 308 e at the base of the thumb, nearest the innerside wall of the side member 304, 306 on which the thumb 308 is mounted,in various embodiments enable misalignment to between the end effector300 and the tray(s) being grasped to be corrected and/or alignmentrefined. For example, in a picking episode, a thumb of the passive sidemember 304 may be moved into position near a handle or other hole on oneside of the tray to be grasped. The convex surfaces 308 a-d may be used,under force control, to slide the thumb partway into the hole. The flatsurfaces 308 e near the base of the thumb may be used to better alignthe passive side with the tray prior to closing the active side member306.

Referring further to FIG. 3A, in the example shown end effector 300includes a force sensor 310 mounted on lateral member 302 and a bracket312 to attach the end effector 300 to a robotic arm. In someembodiments, end effector 300 may be attached to a robotic arm via a pininserted through a hole in bracket 312, enabling the end effector 300 toswing freely and/or be rotated under robotic control, e.g., using one ormore motors, about a longitudinal axis of the pin. In variousembodiments, force sensor 310 detects forces/moments experienced by endeffector 300 in an x, y, and/or z direction. Force sensor 310 may have asingle axis overload of force in the x or y direction (e.g., F_(xy)) ofat least ±10000 N and/or a single axis overload of force in the zdirection of at least ±30000 N (e.g., F_(z)). Force sensor 310 may havea single axis overload of torque in the x or y direction (e.g., T_(xy))of at least ±1000 Nm and/or a single axis overload of torque in the zdirection of up to at least ±1000 Nm (e.g., T_(z)). In some embodiments,force sensor 310 has a single axis overload of force in the x or ydirection (e.g., F_(xy)) of about ±18000 N and/or a single axis overloadof force in the z direction of about ±48000 N (e.g., F_(z)); and asingle axis overload of torque in the x or y direction (e.g., T_(xy)) ofabout ±1700 Nm and/or a single axis overload of torque in the zdirection of about ±1900 Nm (e.g., T_(z)).

In various embodiments, passive side member 304 is fixedly mounted tolateral member 302. The fixed mounting of the passive side member 304may enable forces and moments acting on end effector 300 (e.g., onpassive side member 304) to propagate through the frame of the endeffector (e.g., lateral member 302 and passive side member 304) to forcesensor 310. For example, the fixed mounting of the passive side member304 avoids forces and movements from translating into a movement ofother parts of the end effector such as active member 306 when activemember 306 is being actuated to move thumb 308 to engage with a trayhandle (e.g., to insert thumb 308 into the tray handle).

FIG. 3B is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. In the state shown in FIG. 3B,the active side member 306 has been opened to the open position, e.g.,by a pneumatic or hydraulic piston, motor, or other motive force andstructure housed in lateral member 302 (not shown) in FIG. 3B.

Vector/direction 314 illustrates an example of a closed position (e.g.,the closed position threshold). In various embodiments, the closedposition is a configuration according to which active side member 306forms a normal vector relative to lateral member 302. For example, theclosed position threshold is 90 degrees (or substantially 90 degrees)relative to a direction along which lateral member 302 extends. Asillustrated in FIG. 3B, active side member 306 is moved to an openposition. As active side member 306 is moved to the open position, anangle between active side member 306 and vector/direction 314 isrepresented as angle 313. According to various embodiments, the openposition threshold corresponds to a configuration at which angle 313 isbetween 35 degrees and 50 degrees. In some embodiments, open positionthreshold corresponds to a configuration at which angle 313 is between40 degrees and 50 degrees. In some embodiments, open position thresholdcorresponds to a configuration at which angle 313 is between about 40degrees and about 45 degrees.

In various embodiments, robotic system controls active side member 306(e.g., controls an actuation device to move active side member 306)based at least in part on information obtained by one or more sensors,such as a sensor(s) comprised in active side member 306 (e.g., thumb 308of active side member 306), a sensor(s) comprised in passive side member304 (e.g., a thumb of passive side member), a camera or other sensorcomprised on or around the robot to which end effector 300 is connected(e.g., to capture information pertaining to the workspace of the robot),and the like, or any combination thereof. Active side member 306 iscontrolled according to a plan to grasp, move, and/or place a set of oneor more trays and the information obtained from the one or more sensors.Active side member 306 may be further controlled according to obstacleswithin the workspace of the robot such as another stack of trays (e.g.,an adjacent stack), another robot working to remove a tray another stackof trays (or of the same tray).

In various embodiments, tray pick operations as disclosed herein aresmooth, gentle, and precise and are tolerant to uncertainty anddisturbances. In various embodiments, a pick episode includes one ormore of:

-   -   Lowering to a target pose (adjacent tray) from the hover pose        (above stack) coupled with height checks to refine estimation of        where the tray handle is and a dynamic goal adjustment.    -   Using the active side surface of the end effector to control any        uncertainties in the direction of the rail be it a misplaced        tray or a human error. In various embodiments, after moving to        the hover pose, the robot lowers into the position to align with        the handle, and while it does this lowering motion, force        control is used to ensure that the alignment in the rail        direction is perfect (or substantially perfect). This can be        very likely because the gripper almost perfectly fits the length        of the tray in between itself, and any misalignment can lead to        contact. This contact is ensured to be on the active side panel,        which has a diagonal plane—meaning that the robotic system can        effectively use the contact between the gripper and the        [misaligned] trays to adjust our position using force control.    -   Proceeding to use a three degree of freedom (3 DOF) force        controller (e.g., based on sensor readings from force sensor        310) to find the position of the (tray handle) slot on the        passive side and insert the passive side thumb into the slot        using the convexity of the thumb (e.g., one or more of surfaces        308 a-d, depending on which engage with the tray). In some        embodiments, a 6DOF controller is used to perform XYZ force        control to ensure that the thumb is inserted and XYZ axis moment        to ensure that the plane of the passive side panel is flush        against the plane of the tray outer surface. In some        embodiments, one or more sensors in passive side member 304 (or        in the thumb of passive side member 304) are used to obtain        information associated with a location of the tray, such as        information indicating a position of the second side member        relative to the first tray, information indicative of when the        first tray is in a position at which the end effector is        controlled to engage the passive-side structure with the hole        (e.g., to detect when the tray is in proximity of the tray such        as at an entry of the gripper such as for detection that end        effector 300 is properly positioned to begin a process of        engaging the tray with passive side member 304, etc.), the        recess, or the handle comprised in the first structure,        information indicative of when the first tray is in a position        at which the passive-side structure is engaged with the hole,        the recess, or the handle comprised in the first structure, and        the like, or any combination thereof.    -   Using the flat extremities (e.g., 308 e) of the thumbs to adjust        for any orientation mismatch.    -   When all is good (e.g., in response to a determination that the        passive side member 304 and/or active side member is positioned        properly to grasp the tray, etc.), close the active side (e.g.,        306) with force/moment control, to account for any residual        orientation or positional uncertainty in the tray pose, and lift        the tray up to make sanity checks for the quality of the grasp        (e.g., weight as expected, forces and moments balanced and        otherwise consistent with good grasp). In some embodiments, when        the state of the gripper is deemed good, the active side is        closed with force/moment control enabled in order to refine and        correct for any residual orientation/position errors, which        ensure gentle handling of the tray.    -   The robot safely aborts the pick if it detects any anomalies        related to weight or quality of the trays in the stack or the        quality of the stacking itself.

FIG. 3C is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. FIG. 3D is a diagram illustratingan embodiment of a robotically controlled tray handling end effector. Inthe examples illustrated in FIGS. 3C and 3D, end effector 300 comprisesa thumb 308 on active side member and thumb 320 on passive side member304. Thumb 308 and/or thumb 320 is a structure that is comprised on, orattached to, the corresponding side member. Thumb 308 and/or thumb 320is respectively used to engage a side of a tray, such as a hole, arecess, or a handle comprised in the first structure.

In various embodiments, thumb 308 of active side member 306 and thumb320 of passive side member 304 comprise different profiles. For example,thumb 308 and thumb 320 have different sizes or shapes. Differentprofiles are implemented for thumbs 308 and 320 based on theirrespective uses/purposes in connection with grasping or releasing atray. The robotic system controls end effector 300 to engage a tray(e.g., a hole, a recess, or a handle on a side of the tray) first, andthereafter controls to actuate active side member 306 to position thumb308 to engage the tray (e.g., a hole, a recess, or a handle on anopposing side of the tray). As an example, because thumb 320 ispositioned to engage tray before engaging tray with active side member306, the robotic system has less degrees of freedom (or less ability tomove) active side member 306 or thumb 308 to engage the tray (e.g., inthe hole, the recess, or the handle on an opposing side of the tray).Accordingly, thumb 308 has a profile that is conducive to insertion ofthumb 308 to engage a structure of the tray (e.g., the hole, recess, orhandle, etc.). According to various embodiments, the profile of thumb308 on active side member 306 is steeper than the profile of thumb 320on passive side member 304. In some embodiments, a height of thumb 308is between 25% and 50% larger than a height of thumb 320. In someembodiments, a height of thumb 308 is between 30% and 40% larger than aheight of thumb 320. The width of thumbs 308 and 320 may be the same (orotherwise selected based on the structure of a side of the tray withwhich the thumbs engage, such as a shape of a handle on a side of atray, etc.).

FIG. 3E is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. As illustrated in FIG. 3E, aprofile of a thumb 350 of for a passive side member is contrasted with aprofile of a thumb 360 for an active side member. Height 364 of thumb360 is larger than height 354 of thumb 350. In some embodiments, a ratioof height 364 to height 354 is about 1.3:1. In some embodiments, a ratioof height 364 to height 354 is about 1.38:1. The ratio of height 364 toheight 354 may be between 1.25:1 and 1.5:1. In some embodiments, height354 of thumb 350 is about 1 inch, and height 364 of thumb 360 is about1.378 inches. The difference between heights 364 and 354 allows forthumb 360 to have a steeper cone shape than thumb 350. As an example,widths and/or lengths of thumbs 350 and 360 are the same). The steeperprofile 362 (e.g., steeper cone shape) of thumb 360 (e.g., compared toprofile 352 of thumb 35) causes less surface area with which to engagethe tray (e.g., the structure on a side of the tray). The lesser surfacearea may have a relatively reduced friction thereby facilitating easierinsertion of thumb 360 to the structure of the tray as the active sidemember closes (e.g., is moved to the corresponding closed position). Inaddition, the steeper profile 362 of thumb 360 may permit easierinsertion of thumb 360 as active side member is moved into position toengage the tray (e.g., the tip of thumb 308 takes up less surface areaand thus can more easily fit into the structure of the tray, such as ahandle, even if active side member is mis-aligned with tray).Conversely, the less steep profile of thumb 350 enables a relativelylarger surface area to engage the tray thereby facilitating a bettergrasp of the tray by end effector (e.g., the relatively larger surfacearea of thumb 350 engaging the tray provides better support for the trayand a greater friction between the tray and thumb 350, which may thusprevent slippage of the tray).

Returning to FIG. 3C, thumb 308 of active side member 306 may have oneor more sensors. For example, as illustrated in FIG. 3C, thumb 308comprises sensor 315 a and sensor 315 b on a side of the thumb. In someembodiments, sensor 315 a and/or 315 b are light sensors, such asreflective sensors. For example, sensors 315 a and 315 b mayrespectively emit a light and detect whether the light is reflected backrespectively to sensors 315 a and 315 b. In the absence of a structurein proximity to sensors 315 a and 315 b light may not be reflected backto sensors 315 a and 315 b (or less than a threshold amount of light isdetected by sensors 315 a and 315 b). In contrast, if thumb 308 isinserted to tray or in proximity of the tray (e.g., engaging a hole,recess, or handle of the tray), sensors 315 a and 315 b detect theproximity of the tray. Sensors 315 a and 315 b may be used to determinethat thumb 308 is engaged with the tray. Although FIG. 3C illustratesthumb 308 as comprising two sensors, a various number of sensors may beimplemented.

In some embodiments, sensors 315 a and/or 315 b are configured to have anarrow focus. For example, a standard sensor may be modified with acover that restricts a size of an opening (e.g., a surface area) fromwhich light is emitted and/or received. The narrow focus of sensors 315a and/or 315 b reduces detection of structure/objects at a distance. Forexample, the narrow focus of sensors 315 a and/or 315 b ensures thatsensors 315 a and/or 315 b primarily (e.g., only) detect the presence ofa structure (e.g., a tray) in close proximity. Decreasing a surface areaof an emitter and/or receiver of a sensor has in turn decreasedeffective range of a reflective sensor. As an example, the angle ofincident light reaching a receiver's surface is closer to normal (e.g.,because light rays of larger angles or less normal are trapped in thereceiver's aperture tunnel length. In some embodiments, sensors 315 aand/or 315 b are configured to detect the presence of an object (e.g.,the tray) at a range of about 1 mm. In some embodiments, sensors 315 aand/or 315 b are configured to detect the presence of an object (e.g.,the tray) at a range of up to 3-5 mm. In some embodiments, sensors 315 aand/or 315 b are configured to detect the presence of an object (e.g.,the tray) at a range of up to 1 cm. In some embodiments, sensors 315 aand/or 315 b are configured to detect the presence of an object (e.g.,the tray) at a range of up to 3 cm.

According to various embodiments, one or more sensors comprised on thumb308 (or on active side member 306 in proximity to thumb 308) obtaininformation indicative of when the tray is in a position at which thethumb 308 is engaged with the hole, the recess, or the handle comprisedin the first structure. The robotic system uses the information inconnection with controlling the robot (e.g., robot arm) to position endeffector 300 or to actuate/move active side member 306 to grasp/releasea tray. For example, in connection with grasping a tray, robotic systemuses the information obtained by sensor 315 a and/or 315 b to detectwhether thumb 308 has engaged the tray (and whether the tray is graspedby end effector), and in response to determining that thumb 308 hasengaged the tray the robotic system may determine to move the tray(e.g., to pick up the tray and move the tray to a destination location).The robotic system may use the information obtained by sensor 315 aand/or 315 b in conjunction with information obtained by force sensor310 to determine that the tray is securely grasped before the roboticsystem determines to begin moving the tray.

FIG. 4A is a flow diagram illustrating an embodiment of a process forgrasping one or more tray using a robotically controlled tray handlingend effector. In various embodiments, process 400 of FIG. 4A isperformed by a control computer, such as control computer 128 of FIG.1A, e.g., to control a single-robot tray handling robot, such as roboticarms 112, 114 of FIGS. 1A and 1B and/or a robot equipped with a trayhandling end effector, such as end effector 300 of FIGS. 3A-3E. In theexample shown, at 402 a passive side member (“arm”) is moved to aposition near a grasp hold or other hole or recess in a side of the trayto be grasped. The active side member may be placed in an open positionto facilitate moving the tray handling end effector into positionwithout having to raise the end effector to as great a height, e.g., toclear the top of the source stack. At 404, force control is used to slotthe passive side thumb (or similar protrusion or structure) into thetray hole (or recess). The insertion may include phases, includinginitially using convex surfaces of the thumb to slide the thumb into thehole or recess and then using force control to further and/or more fullyinsert the thumb into the hole or recess, including by sliding flatsurfaces of the thumb into the hole or recess to ensure the end effectoris fully aligned with the tray. At 406, the active side member is closedand force control is used to slot the thumb (or similar structure) ofthe active side member into the hole or recess on the tray on the sideopposite from the passive side member. At 408, the robot tests its graspof the tray, and if the grasp is determined at 410 to be secure therobot moves the tray to its destination at 412. If the grasp isdetermined at 410 not to be secure, the grasp is adjusted at 414 andtested again at 408. For example, the robot may set the tray back downon the source stack, release the tray, and attempt a new grasp. Or, therobot may set the tray at least partly on the source stack and attemptto adjust its grip without fully releasing the tray, e.g., by usingforce control to try to slot the passive and/or active side thumbs,respectively, more fully into the tray.

In various embodiments, a single robot tray gripping robot as disclosedherein uses a vertical workspace of 0-2.7 m, which allows it to pickvery tall stacks and very low stacks while avoiding obstacles andsingularity. The robot can reset itself from any given position andorientation safely and without collision using a set of tools tocalculate its best reset strategy. On a given pick episode the robotwill first analyze the pose of its target, confirm the target and pose,and then plan the best way to safely get to the target, using a directparadigm that uses a combination of collision avoidance and orientationinterpolation coupled with a positional trajectory if the robot isphysically able to clear the tray and move to a hover position above it.If the robot recognizes that it is not able to clear tall stacks due torobot workspace limitations, it will plan a series of joint motions toposition itself directly into the insertion position without needing tohover over it first.

In various embodiments, after executing checks to ensure a successfuland secure grasp, the robot moves to clear the top tray remaining on thesource stack using a set of rail motions orientation interpolations andcollision avoidance and proceeds along a planned trajectory to thevicinity of the destination stack, e.g., a “hover” position over the topof the destination stack.

FIG. 4B is a flow diagram illustrating an embodiment of a process forgrasping one or more tray using a robotically controlled tray handlingend effector. Process 425 is implemented in connection with grasping anitem such as a tray. For example, process 425 is implemented inconnection with 404 of process 400.

According to various embodiments, passive side member (e.g., passiveside member 304 of end effector 300) comprises one or more sensors. Theone or more sensors comprised on the passive side member are configuredto obtain information pertaining to a location of a structure (e.g., atray) in relation to a position of the end effector (or specifically thepassive side member). Examples of information obtained by the one ormore sensors include (i) obtain information indicative of when the tray(or end effector relative to the tray) is in a position at which the endeffector is controlled to engage at least one side of the end effector(e.g., a passive member or a structure comprised on the passive member)with a hole, a recess, or the a comprised in a side of a tray (e.g., atray being grasped), (ii) obtain information indicative of when the tray(or end effector relative to the tray) is in a position at which the endeffector (e.g., a passive member or a structure comprised on the passivemember), etc. The robotic system may use information obtained from theone or more sensors in connection with positioning the passive sidemember (or the end effector generally). In some embodiments, the roboticsystem uses the information obtained from the one or more sensors inconjunction with information obtained from a force sensor to control theend effector (e.g., a thumb comprised on passive side member to engagethe tray).

In various embodiments, the end effector comprises a first sensor thatis configured to obtain information indicative of when the tray is in aposition at which the end effector is controlled to engage apassive-side structure (e.g., a thumb disposed on passive side member)with a hole, the recess, or the handle comprised in a structure of thetray. The first sensor is disposed on the passive side member, such asat or near a distal end of the passive side member (e.g., near thebottom or distal end of a fin of the passive side member). As the endeffector is moved in proximity to the tray, the robotic system usesinformation obtained from the first sensor in connection with moving theend effector to engage the tray with the structure on passive-sidestructure (e.g., a thumb disposed on passive side member). For example,the robotic system uses the information obtained from the first sensorto coarsely position the end effector (e.g., to determine whether thetray is between the side members of the end effector, etc.).

In various embodiments, the end effector comprises a second sensor thatis configured to obtain information indicative of when the tray is in aposition at which the passive-side structure is engaged with the hole,the recess, or the handle comprised in the structure (e.g., a structureon the side of the tray) The second sensor is disposed on the passiveside member, such as in proximity of a structure on the passive sidemember (e.g., near a thumb of the passive side member or near the top ofa fin of the passive side member). As the end effector is moved inproximity to the tray, the robotic system uses information obtained fromthe second sensor in connection with moving the end effector to engagethe tray with the structure on passive-side structure (e.g., a thumbdisposed on passive side member). For example, the robotic system usesthe information obtained from the second sensor to fine tune apositioning of the end effector.

At 426, information is obtained from a first sensor (s). The informationindicates whether a passive arm (passive side member) of the endeffector is in proximity of the tray.

At 428, the robotic system determines whether to engage the passive arm.For example, the robotic system uses the information obtained from thefirst sensor to determine whether to engage the tray with the passivearm. The robotic system determines to engage the tray with the passivearm in response to determining that the plan for moving trays indicatesthat the tray is to be picked and placed in a destination location, andthat the tray is in proximity to the end effector. In some embodiments,the system uses the information obtained from the first sensor todetermine whether the tray is between the passive arm and the active armof the end effector.

In response to determining that the passive arm is not to be engaged(e.g., with the tray) at 428, process 425 proceeds to 430 at which theposition of the passive arm is adjusted. The robotic system controls therobotic arm to move the end effector such as closer to the tray.Thereafter, process 425 returns to 426.

In response to determining that the passive arm is to be engaged (e.g.,with the tray) at 428, process 425 proceeds to 432 at which the roboticsystem uses force control to engage the passive arm thumb with the tray.For example, the robotic system uses force control to engage thumb ofthe passive arm into a structure of the tray (e.g., a hole, recess,handle, etc.).

At 434, information is obtained from a second sensor(s). The informationindicates whether the passive arm thumb is engaged with the structure ofthe tray.

At 436, the robotic system determines whether the thumb of the passivearm is engaged with the tray.

In response to determining that the passive arm thumb is not engagedwith the tray at 436, process 425 returns to 432 and 432-436 arerepeated. For example, the robotic system further controls to move theend effector to engage the structure of the tray with the passive armthumb.

In response to determining that the passive arm thumb is engaged withthe tray at 436, process 425 proceeds to 438 at which an indication thatthe passive arm is engaged with the tray. For example, in the case thatprocess 425 is invoked by 404 of process 400, 428 provides an indicationto the robotic system (e.g., a process running on the robotic system)that the passive thumb arm is engaged with the tray and that process 400may proceed to 406.

FIG. 4C is a flow diagram illustrating an embodiment of a process forgrasping one or more tray using a robotically controlled tray handlingend effector. Process 450 is implemented in connection with grasping anitem such as a tray. For example, process 450 is implemented inconnection with 406 of process 400.

According to various embodiments, end effector (e.g., the lateral memberor active arm) comprises one or more sensors that obtain informationpertaining to a position of the active arm (e.g., active side member 306of end effector 306). In some embodiments, the end effector (e.g., thelateral member or active arm) comprises a sensor(s) that detects whetherthe active arm is in an open position or a closed position. For example,the sensor is a mechanical limit switch that is configured to obtaininformation indicative of whether the active side member is in an openposition or a closed position. In some embodiments, the end effector(e.g., the lateral member or active arm) comprises a sensor(s) thatdetects an extent to which the active arm is in an open position or aclosed position (e.g., the sensor determines a particular orientation ofthe active arm or a particular location of the active arm between theopen position and the closed position, inclusive). For example, thesensor is a light sensor. The light sensor is configured to obtaininformation indicative of whether the active side member is in an openposition or a closed position. The light sensor may be furtherconfigured to obtain information indicative of an extent to which theactive side member is open (e.g., whether the active arm is partiallyopen such as half-way between the open position and the closed position,etc.). The robotic system uses the one or more sensors (e.g., thesensor(s) that obtains information pertaining to a position of theactive arm) to control actuation of an actuator that moves the activearm to move (e.g., to move the active arm between the closed positionand open position).

In some embodiments, end effector (e.g., the active arm of the endeffector) comprises one or more sensors that are used to detect whetherthe active arm (e.g., a thumb of the active arm) is engaged with a tray(e.g., a structure on the tray such as a hole, a recess, or a handle,etc.). For example, the end effector comprises sensor 315 a and/or 315 bof end effector 300 illustrated in FIG. 3C.

At 452, information indicating whether the active arm is open/closed isobtained from a sensor (e.g., a sensor comprised on the end effectorsuch as at the lateral member or active member). The robotic system usesthe sensor to obtain information pertaining to a position of the activearm.

At 454, the robotic system determines whether the active arm is in anopen position. For example, the robotic system determines whether theactive arm is fully open (e.g., opened to the open position threshold).As another example, the robotic system determines whether the active armis sufficiently open to grasp a tray (e.g., if an adjacent stackprevents/restricts the robotic system from fully opening the activearm).

In response to determining that the active arm is not in an openposition at 454, process 450 proceeds to 456 at which the position ofthe active arm is adjusted. For example, the position of the active armis adjusted to permit the end effector to grasp the tray (e.g., toensure clearance of the tray as the end effector is controlled to graspthe tray). The robotic system controls the robotic arm to move theactive arm to further open the active arm, or to fully open the activearm. Thereafter, process 450 returns to 452.

In response to determining that the active arm is in an open position at454, process 450 proceeds to 458 at which the robotic system determinesto engage the tray. For example, the robotic system determines tocontrol the actuator to move the active arm to engage the tray with theactive arm (e.g., a structure on the active arm such as an active armthumb).

At 460, force control is used to adjust the configuration of the activearm to the closed position. In response to determining to engage thetray, the robotic system controls the actuator to move the active arm tothe closed position.

At 462, information is obtained from sensor(s), the informationindicative of whether the active arm thumb is engaged with a structureof the tray. For example, the robotic system obtains information fromsensor 315 a and/or 315 b of end effector 300 illustrated in FIG. 3C.

At 464, a determination of whether the active arm thumb is engaged withthe tray is performed. In some embodiments, the robotic system uses theinformation obtained from the sensor(s) (e.g., the informationindicative of whether the active arm thumb is engaged with a structureof the tray) to determine whether the active arm thumb is engaged withthe tray (e.g., whether the active arm thumb is inserted into a hole,recess, or handle of the tray). In some embodiments, the robotic systemfurther obtains information from a force sensor and uses informationpertaining to forces acting on end effector in connection withdetermining whether the active arm thumb is engaged with the tray.

In response to determining that the active arm thumb is not engaged withthe tray at 464, process 450 returns to 460 to further adjust theconfiguration of the active arm. Process 450 iterates through 460, 462,and 464 until the robotic system determines that the active arm thumb isengaged with the tray.

In response to determining that the active arm thumb is not engaged withthe tray at 464, process 450 proceeds to 466 at which an indication thatthe active arm is engaged with the tray is provided. For example, in thecase that process 450 is invoked by 406 of process 400, 466 provides anindication to the robotic system (e.g., a process running on the roboticsystem) that the active thumb arm is engaged with the tray and thatprocess 400 may proceed to 408.

FIG. 4D is a flow diagram illustrating an embodiment of a process forreleasing one or more tray using a robotically controlled tray handlingend effector. Process 475 is implemented in connection with moving orreleasing an item such as a tray. For example, process 475 isimplemented in connection with 412 of process 400.

At 476, a determination is made to place a tray(s). In some embodiments,the robotic system determines to place a tray in a destination locationin response to moving the tray to an area in proximity to thedestination location at which the tray is to be placed.

At 478, force control is used to adjust configuration of the active armto move the active arm to an open position. In some embodiments, inresponse to determining that the tray is to be placed (e.g., that theend effector is to release) the tray, the robotic system determines tocontrol the active arm to move the active arm to an open position torelease the tray. The robotic system can control the actuator to movethe active arm to the open position threshold (e.g., to move the activearm to a fully open position) or to otherwise move the active arm to anopen position sufficient to enable release of the tray.

At 480, information is obtained from a sensor(s), the informationindicates whether the active arm is open or closed. The robotic systemuses the sensor to obtain information pertaining to a position of theactive arm.

At 482, a determination is made as to whether the active arm is in anopen position. For example, the robotic uses the sensor to determinewhether the active arm is in a fully open position (e.g., that theactive arm is configured at the open position threshold), oralternatively, whether the active ram is sufficiently open to enablerelease of the tray.

In response to determining that the active arm is not in an openposition at 482, process 475 proceeds to 484 at which the configurationof the active arm is adjusted. The robotic system uses force control(e.g., controls the actuator) to move further open the active arm.Process 475 thereafter proceeds to 480 at which 480 and 482 are iterateduntil the active arm is moved to an open position for the tray to bereleased.

At 486, information is obtained from sensor(s), the informationindicative of whether the active arm thumb is engaged with a structureof the tray. For example, the robotic system obtains information fromsensor 315 a and/or 315 b of end effector 300 illustrated in FIG. 3C.

At 488, a determination of whether the active arm thumb is engaged withthe tray is performed. In some embodiments, the robotic system uses theinformation obtained from the sensor(s) (e.g., the informationindicative of whether the active arm thumb is engaged with a structureof the tray) to determine whether the active arm thumb is engaged withthe tray (e.g., whether the active arm thumb is inserted into a hole,recess, or handle of the tray). In some embodiments, the robotic systemfurther obtains information from a force sensor and uses informationpertaining to forces acting on end effector in connection withdetermining whether the active arm thumb is engaged with the tray.

The robotic system obtains the information indicative of whether theactive arm thumb is engaged with a structure of the tray and determineswhether the active arm thumb is engaged with the tray as a check todetermine whether the tray is released or able to be released.

In response to determining that the active arm thumb is still engagedwith the tray at 488, process 475 proceeds to 484 at which the roboticsystem adjusts the configuration of the active arm (e.g., to furtheropen the active arm). Thereafter, process 475 iterates through 480-488until the active arm is disengaged from the tray.

In response to determining that the active arm thumb is not engaged withthe tray at 488, process 475 proceeds to 490 at which an indication thatthe active arm is disengaged with the tray is provided. For example, inthe case that process 475 is invoked by 412 of process 400 of FIG. 4A,490 provides an indication to the robotic system (e.g., a processrunning on the robotic system) that the active thumb arm is disengagedwith the tray (e.g., that the tray is release and successfully moved tothe destination location) and the robotic system can thereafterdetermine whether the stack of trays is complete or that another tray isto be moved.

FIG. 5A is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. In the example shown, endeffector 300 of FIGS. 3A and 3B is shown in a state in which the activeside member 306 is in an open position. In various embodiments, theactivation of the active panel (e.g., active side member 306) isperformed by a cushioned two-way pneumatic cylinder. In the position asshown in FIG. 5A, the end effector 300 is near but not yet immediatelyadjacent to the tray 502, which in this example has handles or holes 504and 506 on opposite sides of the tray 502.

FIG. 5B is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. In the state shown, passive sidemember 304 has engaged the tray 502, e.g., by slotting a thumb ofpassive side member 304 (not shown) into the handle or other hole 506 onthe near side wall of tray 502. For example, a combination of positioncontrol to move the passive side member 304 of end effector 300 to aposition near the hole 506 of tray 502 and force control to slot a thumbor other protrusion or structure of the passive side member 304 into thehole 506 may be used, in various embodiments.

FIG. 5C is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. In the example and state shown,the active side member 306 of end effector 300 has been closed andslotted into hole 504 of tray 502, e.g., by inserting thumb 308 ofactive side member 306 into the hole 504, as shown.

While a single tray is shown being grasped in FIGS. 5A through 5C, invarious embodiments a single robot tray handling robot as disclosedherein may grasp two or more stacked trays, e.g., by grasping abottommost one of them in the manner illustrated in FIGS. 5A through 5C.The robot may then lift and move the stack comprising the tray itgrasped, and any trays stacked on top of the tray it grasped. In variousembodiments, stability is maintained at least in part by the inner facesof the side members (e.g., 304, 306) engaging the outer faces of thetrays in the stack that has been grasped.

FIG. 6A is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. In various embodiments, an endeffector such as end effector 300 of FIGS. 3A and 3B may includestructures as shown in FIG. 6A. In the example shown, end effector 600includes a lateral member 602 and a passive side member 604 having apassive side thumb 605 configured to be inserted into a handle or otherhole or recess on a first side of a tray. The end effector furtherincludes an active side member 606 with an active side thumb 608. Activeside member 606 includes a tab or bracket 610 mounted on the upper partof the inside face of active side member 606, which in this example ispositioned to align with a hole 612 (or set of holes) through lateralmember 602. In some embodiments, the passive side thumb 605 and theactive side thumb 608 have different profiles. For example, the activeside thumb 608 has a steeper curvature or profile than the passive sidethumb 605. As another example, the active side thumb 608 has a largerheight than the passive side thumb 605.

FIG. 6B is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. In various embodiments, an endeffector such as end effector 300 of FIGS. 3A and 3B may includestructures as shown in FIG. 6B. In the example and state shown, endeffector 600 of FIG. 6A is shown in an assembled state (and in a closedposition of the active side member 606). A shoulder bolt 614 (or hingepin, or similar structure) is shown to be inserted through hole(s) 612and tab/bracket 610. A pneumatic or hydraulic cylinder (e.g., cylinder616) is mounted (e.g., by a pivot bracket or other bracket) to an innersurface within lateral member 602.

In various embodiments, cylinder 616 (e.g., a pneumatic cylinder) andend rod 618 comprises a cushioned two-way pneumatic cylinder. Activationof the active panel (e.g., side member 606) is performed by activatingthe cylinder 616. The end rod 618 of the cylinder 616 is connected tothe active side member 606. The active panel, which is rotatablyconnected to the lateral member via a pivot joint formed by insertingshoulder bolt 614 through hole(s) 612 and tab or bracket 610, ispushed/pulled by the pneumatic cylinder to close/open the active sidemember 606, respectively. The actuation of the cylinder 616 iscontrolled, in various embodiments, by a four-way two-position singlesolenoid.

In some embodiments, an end effector such as end effector 600 includes alimit switch mounted near the active side member 606, which ispositioned to be triggered when the active side member is (fully) in theclosed position, as shown in FIG. 6B.

FIG. 6C is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. In the example and state shown,the cylinder 616 has been activated to pull the end rod 618 in,resulting in the active side member 606 being pulled into the openposition, as shown.

In various embodiments, the end effector 600 includes one or moreinfrared (IR) sensors, each comprising an active IR emitter mounted onthe passive side member 604 and a corresponding reflector on the activeside member 606. The IR sensor(s) is/are triggered, in some embodiments,when a tray comes into the reflective field of the sensor as the robotis trying to pick a tray. In some embodiments, two sensors are presentin the passive side member 604: one sensor is located at the extremityof the fin, while the other is placed in the area where a tray would bewhen a tray is engaged by the gripper. A third sensor, in the thumb ofactive side member 606, is directed such that when the active sidemember 606 closes onto the tray, the tray's inner handle area gets inthe view of the sensor thus triggering that the active side member 606has successfully engaged in the tray.

In some embodiments, the lateral member 602 and side members 604, 606are composed of carbon fiber. The lateral member 602 comprises a hollowsquare beam composed of a braided carbon fiber pole that allows fortorsional stiffness. The side panels (e.g., side members 604, 606) arecomposed of ⅜″ carbon fiber plates, machined to specification.

FIG. 6D is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. FIG. 6E is a diagram illustratingan embodiment of a robotically controlled tray handling end effector. Inthe examples illustrated in FIGS. 6D and 6E, end effector 600 furthercomprises a stopping mechanism (e.g., a stopper) 620 and/or a sensor622.

In some embodiments, the robotic system uses sensor 622 to detectwhether the active arm is in an open position or a closed position.Sensor 622 may comprises a set of one or more sensors that are disposedin end effector such as at lateral member 602 and/or active side member606. For example, the sensor is a mechanical limit switch that isconfigured to obtain information indicative of whether the active sidemember 606 is in an open position or a closed position. In someembodiments, sensor 622 is used to detect an extent to which the activeside member 606 is in an open position or a closed position (e.g., thesensor determines a particular orientation of the active side member 606or a particular location of the active side member 606 between the openposition and the closed position, inclusive). For example, sensor 622 isa light sensor. The light sensor is configured to obtain informationindicative of whether the active side member is in an open position or aclosed position. The light sensor may be further configured to obtaininformation indicative of an extent to which the active side member isopen (e.g., whether the active arm is partially open such as half-waybetween the open position and the closed position, etc.). The roboticsystem uses sensor 622 (e.g., the sensor(s) that obtains informationpertaining to a position of the active arm) to control actuation of anactuator that moves the active side member 606 to move (e.g., to movethe active arm between the closed position and open position).

In some embodiments, stopping mechanism 620 is used to restrict activemember from moving past a closed position threshold when the actuator isactuated to move to an active member to the closed position. Thestopping mechanism 620 is configured based on a desired closed positionthreshold. As an example, the closed position threshold can be about 90degrees relative to a plane/vector along which lateral member 602extends in a lengthwise direction, etc.

FIG. 6F is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. FIG. 6G is a diagram illustratingan embodiment of a robotically controlled tray handling end effector. Inthe examples illustrated in FIGS. 6F and 6G, end effector 600 furthercomprises a stopping mechanism 620 or 624 (e.g., a stopper), and/or asensor 622.

In some embodiments, stopping mechanism 624 is used to restrict activemember from opening past an open position threshold when the actuator isactuated to move to an active member to the open position. The stoppingmechanism 624 is configured based on a desired open position threshold.As an example, the open position threshold can be about 130 degreesrelative to a plane/vector along which lateral member 602 extends in alengthwise direction, etc. Stopping mechanism can be coupled (e.g.,bolted, etc.) to lateral member 602 or otherwise integrated with lateralmember 602.

FIG. 7A is a diagram illustrating an embodiment of a sensor for arobotically controlled tray handling end effector. In the exampleillustrated, sensor system 700 comprises a sensor 710 that includes anemitter 720 and a receiver 730. Sensor system 700 may be an opticalsensor that emits rays from emitter 720 (e.g., infrared rays, lightrays, etc.) and that receives rays reflecting from a surface such as atray 740 to be grasped by the end effector. In some embodiments, sensorsystem 700 is implemented as sensor 315 and/or sensor 315 b of endeffector 300 of FIG. 3C.

FIG. 7B is a diagram illustrating an embodiment of a sensor for arobotically controlled tray handling end effector. In the exampleillustrated, sensor system 750 comprises a sensor 710 that includes anemitter 720 and a receiver 730. Sensor 710 further includes a cover 760that reduces the incident light on receiver 730. For example, the range715 of incident light on receiver 730 is smaller than the range ofincident light on a receiver 730 of sensor system 700. As an example,cover 760 can restrict the effective range of degrees by which light canreflect from tray 740 and still be received by emitter 720 (e.g., theangles of incident light are closer to normal for receiver 730 of thesensor system). Sensor system 750 can be used to more precisely detectwhether the end effector is in position to grasp the tray 740. In someembodiments, sensor system 750 is implemented as sensor 315 and/orsensor 315 b of end effector 300 of FIG. 3C.

FIG. 8A is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. In the example shown, trayhandling end effector 800 includes a lateral member 802 and detachableand/or modular passive and active side members 804 and 806,respectively. As shown, the end effector 800 accommodates and is capableof being used to grasp only a single tray.

FIG. 8B is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector. In the example shown, the endeffector 820 has been provided by detaching and removing the sidemembers 807, 806 from lateral member 802 of end effector 800, andinstalling elongated side members 824 and 826. In the configurationshown in FIG. 7B, end effector 820 may be used to pick up at two trayhigh stack of trays—e.g., trays 828 and 830 in the example shown in FIG.8B—for example, by inserting the respective thumbs of the side members824, 826 into holes on opposite sides of the bottommost tray (i.e., tray830 in the example shown).

FIG. 9A is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector equipped with a guide fin. In theexample shown, tray handling end effector 900 includes a lateral member902, passive side member 904, and active side member 906. End effector900 is attached to a robotic arm (not shown) via a force sensor 910 andpivot bracket 912. In the state shown, end effector has in its grasp atray 914. For example, passive and active side thumbs (not shown) ofpassive side 904 and active side member 906, respectively, may beinserted into corresponding holes (not shown) on either side of tray914.

In the example shown, end effector 900 further includes guide fins 918and 920, mounted along the bottom edges of passive side member 904 andactive side member 906, respectively. In various embodiments, the guidefins 918, 920 extend along all or a substantial portion of the bottomedge of the side members 904, 906. As shown, each guide fin 918, 920 hasa shape that flares outward at the bottom, such that the distancebetween the respective bottom edges of the guide fins 918, 920 isgreater than the width of the tray 914 and the distance between theinner faces of the passive and active side members 904, 906 when in theclosed position, as shown.

As shown in FIG. 9A, the end effector 900 is being used to position thetray 914 over tray 916, e.g., to place the tray 914 onto the tray 916.For example, the tray 916 may be the topmost tray on a destinate stackto which the tray 914 is to be added.

In some embodiments, end effector comprises one or more vehicle grippermodules 921 a or 921 b on passive and active side members 904, 906, orguide fins 918, 920. The vehicle gripper modules 921 a or 921 b cancomprise inner surfaces (e.g., surfaces that engage a tray or vehiclesuch as a dolly) that have a relatively higher friction than innersurfaces of passive and active side members 904, 906, or guide fins 918,920. The one or more vehicle gripper modules 921 a or 921 b can beshaped or configured to stably grasp a vehicle (e.g., dolly) when theend effector is controlled to engage the one or more vehicle grippermodules 921 a or 921 b with such a vehicle.

FIG. 9B is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector equipped with a guide fin. In theexample and state shown in FIG. 9B, the guide fins 918, 920 havefacilitated the use of force control to align and place the tray 914onto tray 916, as shown. In various embodiments, the guide fins 918, 920have a certain degree of compliance that facilitates the placing oftrays. The guide fins are designed in various embodiments to haveflexure properties to increase operation speed and tolerances as well asaccuracy.

In various embodiments, tray place episodes by a single robot traygripping robot as disclosed herein are smooth, gentle, and precise andtolerant to uncertainty and wobbling. In various embodiments, a placeepisode includes one or more of:

-   -   Using force control to go down from a hover position above the        destination stack (e.g., as shown in FIG. 9A) and making contact        with the guide fins (e.g., one or both of guide fins 918, 920)        first.    -   The guide fins 918, 920 help to guide the tray being placed into        a position more aligned with the tray on which the tray being        placed is to be place, and the guide fins 918, 920 also provide        feedback signals via the load cell (e.g., force sensor 910) to        adjust the position of the incoming tray on top of the stack.    -   Using force control, the robot makes steady contact with the        destination stack and gently inserts the tray (e.g., 914) onto        it.

FIG. 9C is a diagram illustrating an embodiment of a roboticallycontrolled tray handling end effector equipped with a guide fin. FIG. 9Dis a diagram illustrating an embodiment of a robotically controlled trayhandling end effector equipped with a guide fin. In the examplesillustrated in FIGS. 9C and 9D, end effector 900 further comprisessensor(s) 922 and/or sensor(s) 924. The robotic system uses sensor(s)922 and/or sensor(s) 924 in connection with guiding end effector 900 tograsp tray 914, such as to control end effector 900 to engage tray 914with a structure on the passive side member 90004 (e.g., a thumb on thepassive side member 904).

The robotic system uses sensor(s) 924 to detect whether tray 914 is inproximity to end effector 900 such as in a manner that robotic systemcan finely control movement of end effector to engage the tray with thestructure on the passive side member 904 (e.g., a thumb on the passiveside member 904). In some embodiments, sensor(s) 924 indicative of whentray 914 is in a position at which end effector 900 is controlled toengage the passive-side structure with the hole, the recess, or thehandle comprised in the tray 814.

The robotic system uses sensor(s) 922 to detect whether tray 914 isengaged by passive member (e.g., by the structure on the passive sidemember 904, such as a thumb). In some embodiments, sensor(s) 924indicative of when tray 914 is in a position at which the passive-sidestructure is engaged with the hole, the recess, or the handle comprisedin the tray 914. As illustrated in FIG. 9D, the system determines thatthe robotic system determines that the tray 914 is in a position atwhich the passive-side structure is engaged with the hole, the recess,or the handle comprised in the tray 914 when (i) information obtainedfrom sensor(s) indicates that a structure is adjacent the sensor (e.g.,light is reflected back to sensor(s) 922), (ii) information obtained bysensor(s) 924 indicates that no structure is proximate to sensor(s) 922(e.g., no light is reflected back to sensor(s) 924).

In some embodiments, the robotic system uses information obtained bysensor(s) 922 and/or 924 in connection with determining whether tocontrol the actuator to move the active side member 906 (e.g., to engagetray 914 with a thumb of active side member 906) to grasp tray 914.

FIG. 10 is a flow diagram illustrating an embodiment of an automatedprocess to place one or more trays on a stack. In various embodiments,the process 1000 of FIG. 10 is performed by a control computer, such ascontrol computer 128 of FIG. 1A, configured to control one or moresingle robot tray handling robots, as disclosed herein. In the exampleshown, at 1002 position control is used to position one or more trays tobe placed, e.g., on the top of a destination stack. For example, 3Dcamera and/or other image data may be used to determine a position andorientation of the destination stack, and the robot may be moved (e.g.,along a rail) into a position near the destination stack and then therobotic arm maneuvered to position the tray above the destination stack.At 1004, force control is used to engage the top of the destinationstack. For example, referring to the example shown in FIGS. 9A and 9B,the tray 914 may be lowered until the bottom edge of one or both of theguide fins 918, 920 just touches the tray 916 at the top of thedestination stack. At 1006, force control is used to guide and set thetray (e.g., tray 914) onto the top of the destination stack (e.g., topof tray 916). At 1008, the robot tests to determine whether the traybeing placed is fully and properly aligned with and securely slottedinto the top of the topmost tray of the destination stack.

In various embodiments, the slotting episode (e.g., 1008) serves toensure the stability of the tray on top of the stack and its properinsertion. After the adjustment in the z axis (up/down) and the y axis(axis along rail along with the trays slot, e.g., the axis into and outof the page as shown in FIGS. 9A and 9B) the robot executes a routing toensure that the third direction (x axis, side to side as shown in FIGS.9A and 9B) is also stable. In various embodiments, a slotting episodeincludes:

-   -   Gently pull the tray back on top of the slot as it was placed in        an offset position forward with respect to the stack.    -   Series of force motions to get the tray unstuck and moving        smoothly; if force feedback indicates the tray is over-slotted,        corrective action is performed to reverse the slot. For example,        if the system fails to find a notch at the back of the tray and        there is an equivalent notch at the front of the tray, the        system tries to “slot” against it by reversing the direction of        slotting motion.    -   The quality of the slot is verified by moving the tray forward        and backward and analyzing the resulting force signals. In        various embodiments, the robot learns and/or is manually trained        to recognize force signals indicative of secure (or not secure)        slotting.

If the tray is determined to have been securely slotted (1010), therobot releases the tray (e.g., opens the active side and withdraws thepassive side thumb from the hole into which it was inserted) and theprocess ends. If not (1010), at 1012 the placement is adjusted andtested again at 1008. If after a configured number of attempts the traycannot be verified as having been placed securely, in variousembodiments, the system prompts a human worker to assist it, e.g., byteleoperation or manual work.

FIG. 11 is a diagram illustrating an example of a stack of traysconfigured to be stacked in a specific tray orientation. In the exampleshown, destination stack 1100 includes a tray 1102 stacked on top of atray 1104. A tray 1106 is to be added to the top of the stack 1100. Thetrays 1102, 1104, 1106 each include a pair of dissimilarly shapedrecesses at the top of the tray, e.g., recesses 1110 and 1114 on the topof tray 1102, one a on first side and the other on the opposite side,and corresponding protrusions at the bottom of the tray, e.g.,protrusions 1108 and 1112 at the bottom of tray 1106. As shown in FIG.11, the protrusion 1108 is of a size and shape to fit into recess 1114,while protrusion 1112 is of a size and shape to fit into recess 1110.However, as shown tray 1106 is flipped around such that the protrusion1108 is over recess 1110, which it does not match, while protrusion 1112is positioned over recess 1114, which it also does not match.

In various embodiments, a tray handling robot system as disclosed hereinlearns and/or is trained to recognize a force sensor reading and/orprofile associated with a misalignment as shown in FIG. 11. The systemmay detect, e.g., upon attempting to place the tray 1106 onto tray 1102,that the tray 1106 is not slotted securely onto tray 1102 in a wayassociated with the tray 1106 being flipped around into a reverseposition, as shown in FIG. 11. In various embodiments, in response todetecting an incorrect orientation as shown in FIG. 11, the system liftsthe tray (e.g., 1106) up, rotates the tray 180 degrees around the z(up/down) axis, and renews its attempt to place the tray on top of thedestination stack.

FIG. 12 is a diagram illustrating an embodiment of a tray handlingrobot. In various embodiments, one or more robots such as robot 1200 ofFIG. 12 may be included in a robotic tray handling system as disclosedherein, e.g., robotic arms 112, 114 in FIGS. 1A and 1B. In the exampleshown, robot 1200 includes a robotic arm 1202 and tray handling endeffector mounted on a chassis 1206 (e.g., a carriage, etc.) configuredto be moved under robotic control along rails 1208 and 1210. Asuperstructure comprising vertical supports 1212 and 1214 and upperframe 1216 provides mounting locations for 3D cameras 1218, 1220, 1222,and 1224. In various embodiments, one or more 3D cameras may be placednear the base of the robot.

In various embodiments, robot 1200 is deployed in a tray handling systemas shown in FIGS. 1A and 1B. Source tray stacks may be provided on oneside of rails 1208 and 1210 (e.g., beyond rail 1208 as shown) anddestination tray stacks may be built on an opposite side of rails 1208and 1210 (e.g., on the side of rail 1210 nearer the viewer, as shown).Pairs of cameras on the source tray stack side (e.g., 1218, 1220) anddestination tray stack side (e.g., 1222, 1224) are used to provide aview of relevant portions of the workspace in the vicinity in which therobot 1200 is located and working.

In various embodiments, the image data may be used to do one or more ofthe following: avoid collisions with other robots, tray stacks, andother items present in the workspace; plan trajectories; and positionthe end effector 1204 and/or a tray in the grasp of end effector 1204 inat least an initial position under position control.

In some embodiments, the cameras 1218, 1220, 1222, and 1224 are includedin a vision system used to control of a robotic tray handling system asdisclosed herein. In some embodiments, the vision system is designed toself-calibrate. The robot uses a marker that is installed on one of itsjoints and exposes the marker to the cameras in the system, e.g.,cameras 1218, 1220, 1222, and 1224, which recognize the marker andperform a pose estimation to understand their own pose in worldcoordinates. The robot plans its motion using collision avoidance to getthe marker into a position close to the cameras to get a high-qualitycalibration.

In some embodiments, a onetime manual process follows the automaticcalibration to further ensure the quality of the process. A point cloudis overlaid on top of the simulated graphics of the system and a humanoperator performs the matching or the rendered graphics of the robotplus environment in a simulator to the point cloud as seen by the cameramounted on the robot. Further verification procedures are also in placeto verify perceived depth of objects of known heights in the world frame(coordinates).

In some embodiments, a system as disclosed herein self-calibrates itsown dimensions. The robot moves up and down the rail to find the pickand place locations and uses force control to find the coordinates ofthe input-output slots. It dynamically performs an update. For example,in some embodiments, the system uses specially designed calibrationmotions (including force control) to find the exact locations of each ofthe input and the output facings (where stacks of trays exist) referredto as the “layout”, and updates layout values internally that many timesreveal variations such as uneven ground surfaces, peripheralinstallation misalignments. The robot dynamically performs these updatesthrough its lifespan, in various embodiments.

In some embodiments, the vision system approximates the pose of thetarget tray or the target destination stack to check the robot goalmotions. A vision system scheduler guarantees simultaneous checks whenit is possible to and both input and output targets are in the field ofview.

FIG. 13 is a diagram illustrating an embodiment of a tray handlingrobot. In the example illustrated in FIG. 13, robot 1300 can be mountedon a chassis such as 1206 of FIG. 12, and robot 1300 can be configuredto be moved under robotic control along rails such as rails 1308 and1310 of FIG. 12.

As shown in FIG. 13, robot 1300 includes a robot arm 1302 to which endeffector 1310 is connected. End effector 1310 comprises a passive sidemember 1320 and an active side member 1334, which may be connected via alateral member. Each of passive side member 1320 and an active sidemember 1334 may include one or more structures, such as thumbs, withwhich to engage and grasp a tray. Passive side member 1320 and activeside member 1334 respectively include guide fins 1323 and 1336 thatflare outwards that guides a tray to a position between passive sidemember 1320 and active side member 1334 (e.g., to a position at whichthumbs on the passive side member 1320 and active side member 1334engage the tray). End effector 1310 further comprises a plurality ofsensors, including sensor(s) 1330, sensors 1238 and 1240, and sensors1322 and 1326. As an example, sensor(s) 1330 may correspond to, or besimilar to, sensor 622 of FIGS. 6D and 6E. Sensor 1330 is disposed onthe lateral member of end effector 1310 or otherwise in a position onend effector from which a position of active side member 1334 can bedetected. As another example, sensors 1338 and 1340 may correspond to,or be similar to, sensors 315 a and 315 b of FIG. 3C. Sensors 1338 and1340 are disposed on or around the thumb on the active side member 1334.As another example, sensors 1322 and 1326 may correspond to, or besimilar to, sensor 822 and 824 of FIG. 8C. Sensors 1322 and 1326 aredisposed on or around guide fin 1324 on passive side member 1320.

In various embodiments, end effector 1310 comprises a wire harness 1318with which connections between a control computer of robotic system,such as control computer 128 of FIG. 1A, and the plurality of sensorscomprised in end effector. Wire harness 1318 can provide a singlepathway via which the plurality of sensors are connected to the controlcomputer (e.g., via a network connection). An output pathway (e.g.,cable) connects wire harness 1318 to the control computer. The outputcable of wire harness 1318 can be an M12 connector such as M12-12position sensor cable. For example, each of the plurality of sensors areconnected to a set of one or more input terminals of wire harness 1318.In addition, one or more other components/modules of end effector 1310are connected to the control computer via wire harness 1318. Forexample, an actuator used to cause active member side 1334 is connectedto an input terminal of wire harness 1318. Wire harness 1318 isthereafter connected to control computer such as via a wire/channel thatis connected to, or comprised in, robot arm 1302.

In various embodiments, end effector 1310 comprises force sensor 1314that obtains information pertaining to forces/moments experienced by endeffector 1310 such as by passive side member 1320 and/or active sidemember 1334. Force sensor 1314 is connected to end effector by mountingplate 1316, and the end effector 1310 and force sensor 1314 areconnected to a distal end 1304 of robot arm 1302 by adapter 1312.

In various embodiments, end effector 1310 comprises a cart pickingmodule 133328. Cart picking module 1328 is configured to pick up (e.g.,grasp) a cart such as to enable robot arm 1302 to move a cart from whicha stack of trays are grasped, picked, and placed to destinationlocations. For example, after the cart has been emptied of the stack oftrays, the cart is moved to a predetermined return area (e.g., alocation at which empty carts are stored and recycled for furthertrays). Cart picking module 1328 can be disposed on passive side member1320. The robotic system determines when a cart is empty, and thencontrols the robot arm 1302 to use end effector 1310 to pick up thecart. Thereafter, the control system controls to move the robot arm 1302to place the empty cart in the predetermined return area.

In various embodiments, a single robot tray handling robotic system asdisclosed herein enables destination (output) stacks to be assembledreliably, quickly, and efficiently by picking trays from source stacksand building destination (output) stacks according to a plan, e.g.,based on order, invoice, manifest, or similar information.

Although the foregoing embodiments have been described in connectionwith the grasping, moving, and placing one or more trays, various otherreceptacles or containers may be implemented. Examples of otherreceptacles or containers include bags, boxes, pallets, crates, etc.

Various examples of embodiments described herein are described inconnection with flow diagrams. Although the examples may include certainsteps performed in a particular order, according to various embodiments,various steps may be performed in various orders and/or various stepsmay be combined into a single step or in parallel.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A robotic end effector, comprising: a lateralmember configured to be coupled to a robotic arm; a passive side membercoupled substantially rigidly to the lateral member at a first distalend and configured to engage mechanically with a first recess on a firstside of an object to be grasped; and an active side member coupled tothe lateral member at a second distal end opposite the first distal endand configured to engage mechanically with a second recess on a secondside of the object to be grasped, the second side being opposite thefirst side of the object to be grasped; wherein: the passive side membercomprises a passive-side structure that is configured to be insertedinto the first recess; the active side member comprises an active-sidestructure that is configured to be inserted into the second recess; anda first profile of the passive-side structure is different from a secondprofile of the active-side structure.
 2. The end effector of claim 1,wherein the second profile of the active-side structure is longer thanthe first profile of the passive-side structure.
 3. The end effector ofclaim 1, wherein the second profile of the active-side structure is atleast 30% longer than the first profile of the passive-side structure.4. The end effector of claim 1, wherein the second profile of theactive-side structure has a steeper curvature than the first profile ofthe passive-side structure.
 5. The end effector of claim 1, wherein theactive side member comprises one or more sensors that are used to detectwhether the active-side structure is engaged with a hole, a recess, or ahandle comprised in the object.
 6. The end effector of claim 1, wherein:the passive side member is fixed relative to the lateral member; theactive side member is movable with respect to the lateral member; theactive side member is configured to move between an open position and aclosed position via robotic control; and the open position is between 30and 50 degrees from a normal vector relative to the lateral member. 7.The end effector of claim 6, wherein a processor in communication withthe end effector is configured to operate the end effector in connectionwith an operation to pick one or more objects from a source stack,including by sending to the end effector a signal to place the activeside member in the open position.
 8. The end effector of claim 6,wherein the end effector further comprises a sensor configured to obtaininformation pertaining to a position of the active side member.
 9. Theend effector of claim 8, wherein the sensor is a mechanical limit switchthat is configured to obtain information indicative of whether theactive side member is in an open position or a closed position.
 10. Theend effector of claim 8, wherein the sensor is a light sensor that isconfigured to obtain information indicative of whether the active sidemember is in an open position or a closed position.
 11. The end effectorof claim 10, wherein the light sensor is further configured to obtaininformation indicative of an extent to which the active side member isopen.
 12. The end effector of claim 1, wherein: the passive-sidestructure comprises a first protrusion that is comprised in, or attachedto, the passive side member; and the active-side structure comprises asecond protrusion that is comprised in, or attached to, the active sidemember.
 13. The end effector of claim 12, wherein the end effector iscontrolled to pick an tray at least in part by using position controlbased on image data to move at least the passive side member of the endeffector into a position adjacent to the first structure and to useforce control to insert into the first structure the first protrusion.14. The end effector of claim 12, wherein: the first protrusioncomprises a first thumb structure configured to be inserted into thefirst structure; the second protrusion comprises a second thumbstructure configured to be inserted into the first structure; and a sizeand/or shape of the first thumb structure and the second thumb structureare different.
 15. The end effector of claim 1, wherein the active sidemember comprises: a guide structure comprising a profile that guides afirst object of the one or more objects to be grasped to a position atwhich the active-side structure engages a hole, a recess, or a handlecomprised in the first object.
 16. The end effector of claim 1, wherein:the passive side member comprises a guide structure comprising a profilethat guides a first object of the one or more objects to be grasped to aposition at which the passive-side structure engages a hole, a recess,or a handle comprised in the first object; and the guide structurefurther comprises one or more sensors configured to obtain informationindicating a position of the passive side member relative to the firstobject.
 17. The end effector of claim 16, wherein the one or moresensors comprise: a first sensor that obtains information indicative ofwhen the first object is in a position at which the end effector iscontrolled to engage the passive-side structure with a hole, a recess,or a handle comprised in the first object; and a second sensor thatobtains information indicative of when the first object is in a positionat which the passive-side structure is engaged with a hole, a recess, ora handle comprised in the first object.
 18. The end effector of claim 1,wherein: the active side member is movable with respect to the lateralmember; the active side member is configured to move between an openposition and a closed position via robotic control; and the end effectorcomprises a stopper that prevents the active side member from movingpast a closed position at which the active side member is positioned 90degrees relative to the lateral member.
 19. An autonomous tray handlingrobotic system comprising the robotic end effector of claim 1, whereinthe system further comprises: a memory configured to store dataindicating a set of output stacks to be assembled, each output stackincluding an associated set of objects; and a processor coupled to thememory and configured to control operation of one or more robots, eachconfigured to grasp, move, and place one or more objects at a time,according to a plan, to iteratively pick objects from source stacks ofobjects and assemble the set of output stacks, including by buildingeach output stack by successively placing on the output stack objectspicked from one or more corresponding source stacks; wherein: each ofthe robots comprises a robotic arm and the end effector configured tograsp, move, and place one or more objects without assistance fromanother robot.